PROP-96 - Add Federated ML to Propeller#97
Conversation
1. Status QuoCurrently, Propeller operates on a "Train Once, Deploy Everywhere" model. The intelligence of the system is static and created entirely outside the Propeller runtime environment.
2. The ProblemTo achieve Federated Learning, where devices learn from local data without sharing it, the current architecture faces three blocking issues: A. The Immutable Wasm BarrierThe current Wasm modules are stateless and immutable.
B. The "Dumb" ManagerThe Manager is currently designed as a Task Dispatcher, not a Model Aggregator.
C. The Disconnected Data LoopThere is no mechanism for the Edge device to utilize its own data for self-improvement.
|
c331076 to
06be567
Compare
Use pickle for model2wasm Fix TinyGo entrypoint and m2cgen wrapping workflow Add FL task mode and metadata fields Signed-off-by: JeffMboya <jangina.mboya@gmail.com> Include FL mode and spec in start payload Signed-off-by: JeffMboya <jangina.mboya@gmail.com> Add FL update envelope type for results Signed-off-by: JeffMboya <jangina.mboya@gmail.com> Deduplicate FL helpers; update host and wazero Signed-off-by: JeffMboya <jangina.mboya@gmail.com> Handle train results as FL envelope updates Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com> Coordinate federated rounds; validate and aggregate updates Signed-off-by: JeffMboya <jangina.mboya@gmail.com> Coordinate federated rounds; validate and aggregate updates Signed-off-by: JeffMboya <jangina.mboya@gmail.com> Coordinate federated rounds; validate and aggregate updates Signed-off-by: JeffMboya <jangina.mboya@gmail.com> Coordinate federated rounds; validate and aggregate updates Signed-off-by: JeffMboya <jangina.mboya@gmail.com> Make wazero StopApp idempotent like host Signed-off-by: JeffMboya <jangina.mboya@gmail.com> Include numeric args in wazero WASI argv Signed-off-by: JeffMboya <jangina.mboya@gmail.com> Update runtime and transport logic Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com> Fix CI Signed-off-by: JeffMboya <jangina.mboya@gmail.com> Fix CI Signed-off-by: JeffMboya <jangina.mboya@gmail.com> Fix CI Signed-off-by: JeffMboya <jangina.mboya@gmail.com> Fix CI Signed-off-by: JeffMboya <jangina.mboya@gmail.com> Fix CI Signed-off-by: JeffMboya <jangina.mboya@gmail.com> Fix CI Signed-off-by: JeffMboya <jangina.mboya@gmail.com> Fix CI Signed-off-by: JeffMboya <jangina.mboya@gmail.com> Fix CI Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
examples/fl-train/fl-train.go
Outdated
| @@ -0,0 +1,91 @@ | |||
| //go:build wasm | |||
There was a problem hiding this comment.
Why do we need these build tags and we aren't using them?
There was a problem hiding this comment.
They ensure the file only compiles for WASM targets (GOOS=wasip1 GOARCH=wasm)
TESTING.md
Outdated
| @@ -0,0 +1,89 @@ | |||
| # Testing Guide | |||
There was a problem hiding this comment.
Why do we need this here?
README.md
Outdated
| - ☁️ **Serverless Applications**: Deploy FaaS applications leveraging Propeller's Wasm orchestration capabilities. | ||
| - 🧠 **Federated Machine Learning**: Train machine learning models across distributed edge devices without exposing raw data, perfect for privacy-sensitive applications. | ||
|
|
||
| ## 🤖 Federated Learning |
There was a problem hiding this comment.
This should be done on docs repo
examples/ml-wasm/model_gen.go
Outdated
| package main | ||
|
|
||
| func score(input []float64) float64 { | ||
| return 0.0000000000000008881784197001252 + input[0] * 1.9999999999999996 + input[1] * 2.999999999999999 |
There was a problem hiding this comment.
These seem like magic numbers
| @@ -0,0 +1,14 @@ | |||
| package main | |||
There was a problem hiding this comment.
Can we have a documentation on how to run these two examples?
|
|
||
| script_dir = os.path.dirname(os.path.abspath(__file__)) | ||
|
|
||
| model_path = os.path.join(script_dir, "mymodel.pkl") |
There was a problem hiding this comment.
These two files should be git ignored
- examples/ml-wasm/mymodel.pkl
- examples/ml-wasm/mymodel.wasm
| @@ -0,0 +1,392 @@ | |||
| package manager | |||
There was a problem hiding this comment.
I think we should keep the integration test. It validates the full FL pipeline and ensures components work together correctly.
manager/service_test.go
Outdated
| var weights []float64 | ||
| err = json.Unmarshal(decoded, &weights) | ||
| if err != nil { | ||
| t.Fatalf("Failed to unmarshal weights: %v", err) |
There was a problem hiding this comment.
Let's use supermq strategy
proplet/runtime.go
Outdated
| @@ -0,0 +1,27 @@ | |||
| package proplet | |||
There was a problem hiding this comment.
This should be removed
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
Signed-off-by: JeffMboya <jangina.mboya@gmail.com>
|
This can be closed in favour of #117 |







What type of PR is this?
Feature
What does this do?
Adds ML to WASM demo application
Which issue(s) does this PR fix/relate to?
Have you included tests for your changes?
Yes, manually tested
Did you document any new/modified features?
Yes
Notes